Depth Separations in Neural Networks: What is Actually Being Separated?

Existing depth separation results for constant-depth networks essentially show that certain radial functions in $$\mathbb {R}^d$$ , which can be easily approximated with 3 networks, cannot by 2 even up to constant accuracy, unless their size is exponential d. However, the used demonstrate this are rapidly oscillating, a Lipschitz parameter scaling polynomially dimension d (or equivalently, function, hardness result applies $$\mathcal {O}(1)$$ -Lipschitz only when target accuracy $$\epsilon $$ at most $$\text {poly}(1/d)$$ ). In paper, we study whether such separations might still hold natural setting of functions, does not scale Perhaps surprisingly, answer negative: contrast intuition suggested previous work, it possible approximate 2, {poly}(d)$$ every . We complement showing approximating also {poly}(1/\epsilon )$$ Finally, have polynomial dependence both $$d,1/\epsilon simultaneously. Overall, our indicate order expressing accuracy—if all possible—one would need fundamentally different techniques than existing ones literature.